Metamorphic Testing of Relation Extraction Models

Relation extraction (RE) is a fundamental NLP task that aims to identify relations between some entities regarding given text. RE forms the basis for many advanced tasks, such as question answering and text summarization, thus its quality critical relevant downstream applications. However, evaluating of models non-trivial. On one hand, obtaining ground truth labels individual test inputs tedious even difficult. other there an increasing need understand characteristics in terms various aspects. To mitigate these issues, this study proposes by applying metamorphic testing (MT). A total eight (MRs) are identified based on three categories transformation operations, namely replacement, swap, combination. These MRs encode expected properties different aspects RE. We further apply MT popular models. Our experiments reveal large number prediction failures subject models, confirming effective Further analysis experimental results reveals advantages disadvantages our also uncovers typical issues